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Abstract 


This paper presents findings of an efficacy trial examining the effect of a multi-tiered instruction 
and intervention model on first grade, at risk students’ reading outcomes. Schools (n = 16) were 
randomly assigned to the treatment or control condition. In the fall of Grade 1, students were 
assigned to an instructional tier on the basis of SAT-10 scores (>39th percentile = Tier 1; 16tn- 
39th percentile = Tier 2). In both conditions, Students identified as at-risk (i.e., Tier 2; 1 = 267) 
received 90 minutes of whole group instruction (Tier 1) and an additional 30 minutes of daily 
small group intervention (Tier 2). In the treatment condition, teachers were trained to enhance 
core reading instruction by making instruction more explicit and increasing practice 
opportunities for students in Tier 1. In addition, at risk students were provided an additional 30 
minutes, daily small group intervention with content that was highly aligned with the Tier 1 core 
reading program. Results indicate significant, positive effects of the intervention on students’ 
decoding and fluent reading and substantive effects on total reading achievement. 


An Examination of the Efficacy of a Multi-tiered Intervention in Early Reading 

Considerable research on reading has centered on the prevention of reading difficulties 
and early intervention (Snow, Burns, & Griffin, 1998). A key finding has been that explicit and 
systematic instruction focused on essential skills (e.g., phonemic awareness, alphabetic principle, 
reading in connected text, vocabulary and comprehension; NRP, 2000; NRC, 1998) can prevent 
and remediate early reading difficulties for many students (i.e., estimates range from 92 to 98%; 
Torgesen, 2000). Research also indicates that a range of early reading measures can identify 
students who need early intervention (Deno & Fuchs, 1987; Shinn, 1989). Identifying students 
who need intervention, and successfully teaching these students the early reading skills they 
need, forms the basis of a robust model of prevention and early intervention (Baker, Fien, & 
Baker, 2011; Kame’ enui, Simmons, & Coyne, 2000). 

However, these positive research findings have not found their way into reading practice 
in a way that has made a clear difference in terms of student impact. Patterns on the National 
Assessment of Education Progress (NAEP) support this conclusion. For example, recent NAEP 
findings show that fourth grade reading scores have stagnated over the last few years, and only 
minor increases have occurred over the last two decades (e.g., in 1992, 38% of fourth grade 
students scored in the below basic category; in 2011, the figure was 33%). 

One source of our inability to translate effective research practices into effective 
classroom practices may be the compartmentalized nature of reading intervention research. 
Separate studies in early reading intervention may identify strong effects in discrete reading 
areas such as phonological awareness and phonics, but teachers in schools must put these 
discrete pieces together into an overall system, which may be substantially more difficult than 


sequencing together the findings from separate research studies. There is a noticeable lack of 


research on how separate research-based findings should be integrated into a cohesive system of 
reading instruction and support in public school settings (Gersten et al., 2009; National Joint 
Committee on Learning Disabilities, 2005). The purpose of this paper is to present the results of 
an experiment on the impact of a multi-tiered reading intervention in first grade that addressed 
all major reading components (i.e., phonemic awareness, phonics, fluency, vocabulary and 
comprehension instruction), and was designed to improve the system of instruction used in Tiers 
1 and 2. 
Multi-tiered Instruction as a Conceptual Frame to Operationalize Early Intervention 
Research 

Multi-tiered systems of support represent an attempt to synthesize research on prevention 
and early intervention into a cohesive and comprehensive approach that is empirically grounded. 
Multi-tiered approaches provide reading instruction that increases in intensity within tiers and 
from one tier to the next (Gersten et al., 2009). Students’ tier placement is based on universal 
screening and movement between tiers is presumably based on progress monitoring data. Tier 1 
instruction occurs in general education contexts and is conceptualized as core reading 
instruction. Tier 2 supports offer more intense instruction for students who need re-teaching of 
concepts, intensified explicit instruction, and additional practice with teacher feedback. A 
hallmark of Tier 2 is that instruction is provided in small group formats. The most intense 
instruction is provided in Tier 3, which is characterized by highly explicit, systematic instruction, 
delivered in small group formats or individually. 

Recently the Institute of Education Sciences (IES) released a practice guide on RtI and 
multi-tiered reading instruction and intervention in the primary grades. This practice guide, as 


well as the 17 other IES practice guides, is intended to summarize research in critical areas of 


education practice and provide an organized set of recommendations for practitioners designed to 
guide implementation efforts. The practice guide provides five recommendations for 
implementing multiple tiers of support in school settings (Gersten, et al., 2009). Each 
recommendation is rated according to the strength of scientific evidence supporting its 
effectiveness. Evidence ratings occur at three levels. A strong rating implies there is consistent 
and generalizable evidence linking the practice to improved outcomes. A moderate rating means 
either there is strong causal evidence that may not be generalizable to the population addressed 
in the recommendation, or that there is not strong evidence available for causal attributions. A 
low rating means that empirical evidence is meager and the recommendation is based primarily 
on the panel’s expert opinion. Given the focus of the current study is on testing a multi-tiered 
framework of support in Tiers | and 2, we briefly describe the four practice guide 
recommendations for Tiers | and 2. 

Tier 1. The practice guide includes two recommendations for Tier 1. The first is to screen 
all students for potential reading problems at the beginning and middle of the year and in 
between these assessments, and following the middle of the year assessments, to regularly 
monitor the progress of students at elevated risk for reading difficulty. In the early grades (K-2), 
appropriate target areas are letter naming fluency, phoneme segmentation, pseudoword reading, 
word identification, and oral reading fluency. The evidence for this recommendation was 
moderate, based on five correlational studies demonstrating these measures are associated with 
later reading performance (see Compton, Fuchs, Fuchs, & Bryant, 2006; McCardle, 
Scarborough, & Catts; 2001; O’Connor & Jenkins, 1999; Scarborough, 1998; Speece, Mills, 


Ritchey, & Hillman, 2003). 


The second practice guide recommendation is to provide time for differentiated 
instruction within Tier 1 based on assessment data. The evidence for this recommendation was 
low, based on one study meeting WWC standards with reservations (see Connor, et al., 2009). In 
this study, 10 schools used student data to group students for instruction. Schools in the 
intervention condition received professional development on how to individualize instruction 
using recommendations and strategies provided by software program. Correlational results 
indicated that there was an association between fidelity and outcomes. Student growth was 
higher in classrooms where teachers implemented the program with greater fidelity (Gersten et 
al., 2009). 

The concept of Tier 1 differentiation is important in the current study. An important 
purpose of differentiation is to make core reading content accessible to students at the lower end 
of the Tier 1 spectrum. In other words, differentiation can result in more effective instruction for 
low Tier 1 students. Another way to achieve the same differentiation principle is to expand the 
overall quality of Tier 1 materials and instruction so that all students at the lower end of the Tier 
1 spectrum receive better Tier | instruction. In other words, these students do not have to 
struggle first to receive more intense Tier | instruction. A fundamental purpose of the ECRI 
intervention was to expand the overall quality of Tier | instruction to better meet the needs of 
students at the low end of the Tier | continuum. 

Tier 2. Clearly, much more is known about effective early reading instruction in Tier 2 
than in Tiers 1 or 3. The findings regarding Tier 2 show a high degree of consistency and strong 
consensus on the implications for districts and schools working to translate research to practice. 
The major conclusion of this research is that approximately 30 minutes of small group 


instruction each day (3 to 5 days per week) is highly effective for students who are struggling 


with learning to read. The way instruction is provided is carefully described in these studies, 
which demonstrate consistency in how instruction should be delivered by the teacher (Gersten et 
al., 2009). 

Virtually all recommendations of Tier 2 instruction focus on the fundamental and 
foundational importance of explicit instruction (Gersten et al., 2009). A number of components 
define explicit instruction across these Tier 2 interventions. Essential components are (a) 
teachers model the skills and knowledge they expect students to use and apply (Mathes et al., 
2005), (b) teachers provide multiple practice opportunities for students during the lesson (Gunn, 
Smolkowski, Biglan, Black, & Blair, 2005; Vadasy, Sanders, & Peyton, 2005), (c) teachers 
correct and or address student errors and misunderstandings immediately and systematically 
(Gunn et al., 2005), and (d) teachers pace lessons in a brisk manner so that student engagement is 
high and so that multiple areas of reading development (e.g., phonemic awareness, vocabulary, 
word reading) can be addressed in the 30-minute lesson. 

The purpose of these research studies was to test the effectiveness of Tier 2 
instruction and the vast majority of the studies do not state what the Tier | instructional program 
is, or what is occurring in Tier 1. The Tier 2 programs use separate materials, and the integration 
of Tiers 1 and 2 is not described. Conceptually, however, Tier 2 early reading instruction is 
supposed to directly supplement, not replace, Tier | instruction, and, in most conceptualizations 
of the three-tier model, Tier 2 is provided in addition to Tier 1 instruction (Vaughn, Wanzek, 
Woodruff, & Linan-Thompson, 2007). Thus, it is imperative to examine the effectiveness of Tier 
2 in concert with Tier 1 implementation. In a recent review of 22 empirical studies, Hill et al., 


(2012) concluded that the “lack of data on Tier 1 instruction suggests that researchers who are 


evaluating the efficacy of Tier 2 elementary reading interventions are doing so outside of the 
context of RTI” (p.122). 

They based this conclusion on the fact that of the 22 studies conducted between 2005 and 
2011, not a single study reported fidelity of implementation for core reading instruction (e.g., 
Case et al., 2010; Fuchs et al., 2008; Mathes et al., 2005). There were four exceptions in cases 
where researchers had implemented a supplemental tier 1 intervention in the context of core 
reading instruction and, consequently reported fidelity of implementation for the supplemental 
intervention only, and not for the general core reading portion (Loftus et al., 2010; McMaster et 
al., 2005; Pullen et al., 2010; Schuele et al., 2008). For example, McMaster et al. trained teachers 
to implement Peer-Assisted Learning Strategies (PALS) into the teachers’ Tier | core reading 
program and reported fidelity data for the PALS portion of the intervention, but not for the 
remaining portion of the Tier 1 core reading time. 
Studies of Tier 2 in the context of Multi-tiered Implementation 

Despite the popularity of RtI and multi-tiered intervention models, there are few 
experimental research studies on their effectiveness (Gersten et al., 2009; Hill et al., 2012; IES 
RFA CFDA 84.324A, 2008). While there are numerous studies that examine the efficacy of 
discrete Tier 2 interventions, there is a lack of a research base on the effect of multiple tiers of 
support on student outcomes or studies that even provide data on what is occurring in Tier | (Hill 
et al., 2012). However, there are a few notable exceptions that have clearly established a 
connection between the nature of intervention delivered in Tier 2 and instruction delivered in the 
Tier 1 core reading program (Chambers et al., 2011; Kerins et al., 2011; O’Connor et al., 2005; 


Scanlon et al., 2008). 


Two recent studies warrant attention and are most relevant to the current study in that 
they (a) employed Clustered Randomized Controlled Trials (i.e. randomly assigned schools to 
condition) and (b) directly or indirectly examined the relative effectiveness of variations of Tier 
1 and Tier 2 intervention approaches on student learning outcomes. In one study, Chambers et al 
(2011) examined the relative effects of a newly developed Tier 2 computer-assisted program and 
their standard one-to-one tutoring process. Both Tier 2 interventions were developed by the 
researchers to supplement and be integrated with their Success for All (SFA) Tier 1 core reading 
program, however the authors reported that the computer assisted program was more aligned 
with their SFA core reading program. While the authors found significant differences favoring 
the Tier 2 computer condition on Woodcock Word ID, Word Attack and Passage Comprehension 
for the first grade participants and no significant differences for the second grade participants, 
they concluded that they could serve more at risk students in the highly aligned, integrated Tier 1 
and Tier 2 approach that included small group computer-assisted supports. The authors also 
reasoned that the CAI program was more effective (in first grade) or as effective (in second 
grade) as the one-on-one tutoring because it was more closely aligned with the Tier 1 SFA core 
reading program. 

In another study, Scanlon et al (2008) examined the relative effectiveness of three service 
delivery approaches for schools to provide two tiers of intervention for preventing early reading 
difficulties for at risk kindergarten students. Specifically, they compared a Professional 
Development Only (PDO) condition, an Intervention Only (IO) condition, and a Combined PD 
and Intervention condition. The PD conditions included a 3 day workshop on enhancing core 
reading instruction through Interactive Strategies Approach (ISA). The ISA is not a program; 


rather it is an approach that teaches teachers to best differentiate student supports by better 
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understanding their students’ instructional needs and maximizing instruction through precise 
student grouping. It also enables teachers to provide better instruction through a variety of 
techniques and activities (Scanlon & Sweeney, 2005). 

The Intervention Only condition provided small group instruction in groups of 3 students 
or fewer by research staff twice a week. The intervention was designed as a direct way for 
teachers to provide targeted intervention to at risk students utilizing principles of the Interactive 
Strategy Approach. Thus, it was not a packaged Tier 2 intervention, but a set of techniques that 
interventionists would implement that included “reading books, learning about letters and letter 
sounds, phonemic awareness, and writing” (p352). The Combined Intervention condition 
incorporated the PD from the first condition and the direct intervention components of the 
second condition; therefore it arguably represented a highly aligned Tier 1 and Tier 2 model, 
given ISA approaches supported each instructional tier. In terms of student achievement, the 
authors hypothesized that the combined PD and intervention condition would be the most 
effective to improve student learning relative to the PD or Intervention Only conditions. 
However, results indicated no significant differences across the three conditions for differentially 
reducing the percentage of students at risk for reading difficulties at the end of the school year. 

Thus, across the Chambers et al study and the Scanlon et al study, results were mixed in 
terms of accelerating student learning through integrated and aligned Tier | and Tier 2 
intervention approaches. 

Overview of ECRI Multi-tiered Intervention 

The multi-tiered instruction and intervention model implemented in the ECRI 

intervention incorporates many of the recommendations in the RtI practice guide, including: (a) 


screening all students for reading difficulties at the beginning of the school year; (b) providing 
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time in Tier 1 for differentiated reading instruction for all students based on assessment of 
student’s current reading level; (c) providing intense and systematic instruction in Tier 2 on up to 
three foundational reading skills, and (d) monitoring the progress of Tier 2 students at least once 
per month (Gersten et al., 2009). However, there are key differences between ECRI and current 
models of RtI. First, ECRI increased the level of explicitness of Tier 1 instruction by redesigning 
the core reading program, following principles of direct instruction (Baker, et al., 2010; Coyne, 
Kame’ enui, & Carnine, 2011). Second, increased specificity was achieved by implementing 
daily lesson maps and intervention template protocols that articulated the learning objectives and 
the procedures teachers would use to address these objectives (WRRFTAC, 200#). Third, ECRI 
increased the alignment between Tier | instruction and Tier 2 intervention by implementing 
daily, 30-minute Tier 2 lessons using the same scope and sequence of skills from the core 
reading program. These three principles are described in more detail below. 

Increased Explicitness. Although many schools implementing multi-tiered instruction 
models use core reading programs to guide Tier | instruction and support, we hypothesize that 
many of these programs do not provide the level of explicit instruction or instructional scaffolds 
students at risk of reading difficulty need to be successful (Baker, et al., 2010). Explicit teacher 
modeling is frequently absent, and, if present, the models are vague and inconsistent. The 
practice opportunities necessary for deep understanding are limited in number and do not provide 
enough guided support for struggling readers. The practice opportunities are often restricted in 
range so that students end up practicing what they have learned under a narrow set of examples 
(Baker, Fien, Baker, 2010; Nelson-Walker et al., 2012). The ECRI enhancements were designed 
to make core instruction more explicit and systematic (Baker, et al., 2010). Specifically, core 


reading programs were restructured to include the following explicit instructional elements: (a) 
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learning objectives for the reading block were made more clear for the students, (b) teacher 
models were made more conspicuous through visual models, verbal directions, and fuller and 
clearer explanations; (c) explicit connections were systematically incorporated between new and 
previously learned concepts; (d) the amount of guided and independent practice opportunities 
were greatly increased and were calibrated to facilitate deliberate and highly accurate practice 
conditions for students; and (e) previously learned material was revisited and reviewed 
systematically (Carnine & Kameenui, 1992; Coyne, Kame'enui, & Carnine, 2011). 

The rationale for these changes is that the majority of at-risk students’ reading instruction 
occurs in Tier 1 settings. For example, the RtI practice guide recommends 30 minutes of 
intervention in addition to Tier 1 instruction. Given that Tier | instruction may be 60 to 90 
minutes per day, it makes sense that this instruction should be aligned with the needs of 
struggling students. The ECRI Tier | enhancements maintain the core program materials but 
supplement these materials when necessary, and alter instruction focusing on these materials 
(Kame’enui, et al., 2003; Western Regional Reading First Technical Assistance Center 
[WRRFTAC], 2005) by incorporating instructional design and delivery features that have been 
shown to be efficacious for struggling readers 

Increased Specificity. Our second principle was to increase the specificity for instruction 
and intervention routines. In Tier 1, we did this in two ways. Lesson maps were created to 
enhance the design of existing programs that have already been created (Kame’ enui et al, 2003; 
WRRFTAC, 2005). Lessons maps highlight the most essential components of a lesson, provide 
time and grouping guidance for each component, and recommend deleting or deemphasizing 


parts of a lesson that are not essential (e.g., an arts and craft activity; Baker, et al., 2010). 
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Second, instruction templates enhance the instructional design and delivery features of 
the core program by re-organizing lessons to increase explicitness and instruction intensity 
(WRRFTAC, 2004). The templates provide (a) clear, step by step directions for how to model 
key reading skills and concepts, and (b) guided and structured opportunities for students to 
practice foundational reading skills and to engage in interactive text-based discussions (Baker, et 
al., 2010). Essentially, the instruction templates organize the instructional design elements of a 
lesson to be more explicit and to be more intensive for students that are reading below grade 
level. Additionally, the high degree of specificity allows instructional leaders (i.e., reading 
coaches) to enter a classroom and quickly ascertain if instruction is begin delivered with fidelity. 

Increased Alignment. The final principle is strong alignment of content and instruction 
in Tiers | and 2 (Baker, et al., 2010), to provide a cohesive instruction format within and across 
tiers. Gersten et al. (2009) recommend that schools should adopt supplemental Tier 2 
interventions that are “compatible with their school’s core reading program” (p. 20) but suggest 
that “alignment is not as critical as ensuring that instruction is systematic and explicit” (p. 23). 
We believe both goals are attainable and in concert should lead to student outcomes that are 
greater than either component implemented in isolation (i.e., systematic and explicit instruction 
vs. instruction in Tiers | and 2 that is highly aligned) (Baker, et al., 2010). The ECRI 
intervention incorporates both of these features. 

In the current study we examine the effectiveness of a Tier 2 intervention that is highly 
aligned with Tier | instruction. In fact, we designed the small group lesson based on the scope 
and sequence of skill instruction from the Tier 1 core reading program. We hypothesize that 
learners will benefit from the efficiencies gained by teaching fewer skills more thoroughly across 


both Tier | and Tier 2 (Baker et al., 2010). This prediction is based on strong theory and research 
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demonstrating that deep and redundant learner practice leads to sustained cognitive changes, 
such as permanent lexical representations (Bransford, Brown, & Cocking, 2000; Perfetti, 2007). 
For example, in an aligned intervention, small group instruction might include pre-teaching a 
new sight word in Tier 2 that will be taught the following day in Tier 1. We predict that this will 
be more beneficial for students-at risk for reading difficulty compared to an intervention that 
teaches different content, with different instructional language and routines in Tier 2. 

Purpose of the Study and Research Hypotheses 

Due to the lack of systematic study of how multiple tiers of support function in concert, 
IES initiated the funding priority for the development and evaluation of systemic interventions, 
such as RtI or multi-tiered intervention models (IES RFA CFDA 84.324A, 2008). The purpose 
of this study was to evaluate the efficacy of a highly specified, multi-tiered reading intervention 
to improve the reading achievement of first grade at-risk students relative to a business-as-usual 
multi-tiered condition. 

Our hypothesis is that at-risk readers attending intervention schools would outperform 
students in control schools on a range of reading outcomes. It is important to note that we did not 
test the efficacy of an RtI model; rather, we tested the efficacy of a multi-tiered instruction and 
intervention model. While we used screening data to place students in their respective tiers, and 
we did collect progress monitoring data to analyze student responsiveness, we did not use 
responsiveness data to regroup students or move students across tiers for the entire school year. 
Our rationale was that schools should not use responsiveness data to regroup students until after 
the school can demonstrate high quality implementation of Tier 1 and Tier 2 intervention. In the 


subsequent year of the study, we conducted a follow-up study in which treatment schools utilized 
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responsiveness data to guide intervention decision-making (including regrouping of students 
across and within tiers). 

Our second hypothesis is related to an important part of the instructional environment in 
which students make strong reading growth. This concerns the role of student reading practice, 
particularly in the context of instructional goals related to the development of fluent reading. 
Specifically, we expect that practice that occurs in the context of instruction that is highly 
specified (the intervention condition) will lead to better reading gains than practice in the control 
condition (Ericsson, 2008). We also expect that intervention teachers will provide more 
opportunities for students to practice reading skills than control teachers. Thus, we expect an 
interaction between treatment condition and the rate of student practice opportunities, and the 
value of practice opportunities in the treatment condition to be stronger in terms of their 
association with student outcomes than practice opportunities in the control condition. This 
hypothesis is tied to a rather straightforward proposition that practicing something (i.e., fluent 
reading) with lots of errors may be more harmful than less practice that is more deliberate and 
error free. 

Method 

This study was part of a four-year project designed to examine the efficacy of the 
Enhanced Core Reading Instruction (ECRI) multi-tiered reading intervention in first grade 
classrooms. Project ECRI was a cluster-randomized controlled trial that nested students and 
teachers within schools. In this study, we report results from the first wave of schools that 
participated in the project (2009-2010). 


Participants 
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In the first wave of the project, 16 schools in three school districts were randomly 
assigned to the treatment or comparison condition, resulting in eight schools in each condition. 
Within participating schools 42 first grade teachers participated in the study, including 23 
teachers in the ECRI treatment condition and 19 in the comparison condition. Teachers averaged 
14 years of teaching experience (SD = 8.71). Instructional assistants, employed by districts and 
trained by ECRI staff, provided small group instruction. A total of 267 students at-risk for 
learning disabilities in reading were included in this analysis, 120 in treatment and 147 in the 
control. Of these students, 5.2% received special education services, 3.3% in the treatment 
condition and 6.8% in the comparison condition. A total of 12.0% of students were English 
Language Learners, 12.5% in the treatment and 11.6% in the control. The average percentage of 
first grade Hispanic students across the schools included the sample is 21.38%, and 2.52% first 
grade students were African American students, as reported by the National Center for Education 
Statistics (NCES) in 2009-2010 (nces.ed.gov). A total of 46% were eligible to receive Free and 
Reduced Price Lunch, 43% in the treatment and 49% in the control. Further description of the 
students identified for Tier 1 and Tier 2 is provided in Nelson et al. (2013). 

Recruitment and Assignment Procedures 

In the spring of the year prior to participation in the project, school districts implementing 
a multi-tiered service delivery model for reading instruction were recruited to participate in the 
ECRI project. Principal investigators met with school district administrators, principals, and first 
grade teachers that expressed interest in participating to describe the project and answer 
questions about participation. To be eligible to participate, schools agreed to (a) use a published, 


comprehensive core reading program (identified and adopted through standard district 
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procedures) during a 90-minute reading block for Tier 1, and (b) provide students identified for 
Tier 2 with an additional 30 minutes of small group instruction. 

In the first wave of the project, we recruited 22 schools in three Oregon school districts to 
participate in Project ECRI. Four of these schools elected to not participate in the study because 
of changes in school leadership between recruitment and the beginning of the study. The 
remaining 18 schools were randomly assigned to the treatment or comparison (standard practice) 
condition. We blocked schools on district before random assignment to control for core curricula 
and other important factors, ensuring similar schools in each condition. After random 
assignment, two schools participating in Wave | (one treatment and one comparison school) left 
the project, leaving 16 schools participating in this study. 

Identification of at-risk readers. Students enrolled in first grade classrooms (n = 1,037 
were recruited to participate in the project, 533 in the treatment condition and 504 in the 
comparison condition. Of these students, we obtained parental consent for 969 students, 500 in 
the treatment condition and 469 in the comparison condition. Parents and guardians were 
unaware of the school’s assignment to condition at the time of consent. Fall scores on the reading 
portion of the Stanford Achievement Test-10th edition (SAT10) were used to assign students to 
Tier 1, Tier 2, or Tier 3, although Tier 3 was not a focus in the project. Students that scored 
between the 16th and 39th percentile on the SAT10 in the fall of first grade were assigned to Tier 
2. Students that scored above the 39th percentile on the SAT10 or below the 16tn percentile were 
assigned to Tier 1 and Tier 3, respectively. Of the students for whom we obtained parental 
consent, 870 students participated in the fall assessment (n = 422 treatment, n = 448 


comparison). In total, 360 students were assigned to Tier | (n = 167 treatment, n = 193 
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comparison), 267 students were assigned to Tier 2 (n = 120 treatment, n = 147 comparison), and 
243 students were assigned to Tier 3 (n = 135 treatment, n = 108 comparison). 
Implementation 

Treatment and comparison teachers conducted daily reading instruction using their 
district-adopted, published, comprehensive core program during a 90-minute reading block for 
Tier 1. Districts participating in the study selected to use Houghton Mifflin or Scott Foresman 
core reading programs. Within district, the same core program was used across treatment and 
comparison conditions. Students identified for Tier 2 received an additional 30 minutes of small 
group instruction, delivered by the classroom teacher or an instructional assistant. 

Treatment condition. The ECRI intervention was designed to increase the quality of 
instructional interactions between teachers and students by supporting the implementation of 
high quality explicit instruction. The intervention supported implementation through (a) Tier 1 
enhanced core reading instruction, (b) Tier 2 small group instruction that was highly aligned with 
Tier 1 instruction, and (c) comprehensive professional development. 

Tier 1: Enhanced core reading instruction. The mean Tier | class was 21.5 students (SD 
= 5.1), with a mean of 22.4 students (SD = 4.5) in the treatment condition and a mean of 20.7 
(SD = 5.5) students in the comparison condition. As previously described, we enhanced core 
instruction so that it was more explicit and systematic than would otherwise be provided using 
the core reading program alone (Baker, et al., 2010). To enhance core instruction, as previously 
described, ECRI teachers used lesson maps (WRRFTAC, 2005) that specified content and 
instructional templates that specified explicit teaching routines (WRRFTAC, 2005). Teachers 


reported that Tier 1 classrooms spent an average of 57.5 (SD = 24.8) minutes in whole group 
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instruction, 29.4 (SD = 21.7) minutes in small group instruction and 34.7 (SD = 22.7) minutes in 
independent work during Tier | instruction. 

Tier 2: Small group instruction highly aligned with ECRI Tier 1. District-employed 
instructional assistants led ECRI small groups. Overall, 19 small group instructors participated in 
the study in the treatment condition. All of the small group instructors were female. Of the small 
group instructors in the treatment condition, 17 were instructional assistants, one was a 
classroom teacher and one was a Title I teacher. ECRI funds were used to support small group 
instruction in treatment schools to ensure standardized implementation. Groups met 30 minutes 
daily for approximately 26 weeks between pre- and post-testing. The location where Tier 2 
intervention was provided differed across schools, but was most often held in a reading 
specialists’ room outside of the Tier 1 classroom. 

Tier 2 intervention materials were developed by the project team to be highly aligned 
with core reading program content. Small group instructors used the same instructional routines 
as were used in Tier | to ensure consistent instructional language and delivery across enhanced 
core and intervention instruction. Students receiving Tier 2 intervention were pre-taught content 
in daily, 30-minute lessons in small groups comprised of 3 to 5 students. For example, if the 
sound spelling /o/ was being taught in the next day’s core lesson, then small group instruction 
preceding that lesson would pre-teach sound spelling /o/, along with sound combinations and 
words that would be taught during whole group instruction the next day. Although multi-tiered 
intervention was delivered using a standard protocol approach, teachers and small group 
instructors used data to group students and to refine instruction. For example, the small group 
instructor tracked group and individual errors during instruction and used a portion of the time 


devoted to Tier 2 small group intervention to re-teach content as needed. Because of the focus on 
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pre-teaching and increasing students’ accuracy with core content, we hypothesized that our at- 
risk readers would be more engaged and successful during practice opportunities during the 
whole group instruction. 

The content of small group lessons emphasized foundational reading skills including 
phonemic awareness, decoding, and fluency in reading connected text. Specifically, small group 
lessons targeted irregular word reading (~ 1 minute), phonemic segmentation and blending that 
included the target sound (~ 1 minute), direct teaching of the target sound spelling (~ 2 minutes), 
blending practice and decodable word reading that included the target sound spelling as well as 
previously taught sound spellings (~ 3 minutes), decodable text reading that included words with 
the target sound spelling as well as previously taught sound spellings (~ 15 minutes), dictation 
that included the target sound-spelling (~ 5 minutes), and re-teaching previous content that had 
been challenging for students as evidenced by group and individual error rates (~ 3 minutes). 

Professional development and coaching. Teachers in the treatment condition 
participated in three days of professional development on the ECRI instructional model prior to 
the beginning of the school year and two days of follow-up activities in October, delivered by 
ECRI trainers (i.e., ECRI expert coaches and project staff). Professional development for 
teachers emphasized (a) instructional teaching routines; (b) overview of research on beginning 
reading content and skills including phonemic awareness, phonics, vocabulary, comprehension 
and fluency in reading connected text; and (c) strategies for increasing student engagement in the 
lessons. Small group instructors received two days of professional development in the fall and 
one day in January. Professional development for small group instructors emphasized 
instructional teaching routines and strategies for increasing student engagement in lessons. All 


professional development sessions included instructional delivery demonstrations by the ECRI 
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trainers and opportunities for participants to practice using instructional strategies with feedback 
from the ECRI trainers. 

Teachers and small group instructors also received comprehensive coaching support from 
ECRI project staff through classroom and small group visits once per month and regular study 
group meetings facilitated by an ECRI expert coach. The high degree of specificity utilized in 
lesson plans and instructional templates allowed ECRI coaches to enter a classroom and quickly 
determine if instruction was being delivered as intended. Regular observations allowed coaches 
to provide feedback to teachers regarding their delivery of instruction related to the explicit 
instructional design principles associated with the intervention. ECRI coaches followed a 
coaching protocol to ensure standardization of coaching support across the treatment schools. 

Comparison condition. Comparison schools used the standard, district-adopted core 
reading program for Tier 1 instruction. Comparison teachers reported that Tier 1 classrooms 
spent an average of 36.5 (SD = 19.9) minutes in whole group instruction, 44.3 (SD = 22.3) 
minutes in small group instruction and 28.0 (SD = 18.2) minutes in independent work. 
Comparison schools used a business-as-usual model to deliver Tier 2 small group instruction 
(i.e., they did not use the ECRI Tier 2 intervention materials), which may have included re- 
grouping students across tiers based on progress monitoring data. Teachers in the comparison 
condition reported that Tier 2 intervention materials included a range of supplemental and 
intervention materials, including published, standardized protocol intervention materials as well 
as teacher-developed materials. 

In the comparison condition, the school district scheduled and provided standard 
professional development activities. 80% of teachers reported that someone in their school 


provided them with instructional support, guidance and coaching about teaching students how to 
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read. Teachers that indicated they received support reported that they received an average of 6.0 
(SD = 12.7) hours per month of this type of coaching. Teachers in the comparison condition 
reported receiving 1.7 (SD = 1.6) days of professional development directly related to reading 
instruction across the school year, on average. 

Fidelity of implementation. Observations of implementation fidelity conducted by 
trained data collectors indicated nearly all treatment teachers used lesson maps (M = .90) and 
instructional templates (M = .91) during instruction, and instruction in treatment classrooms was 
consistent with the intent of the lesson maps and templates to improve the quality of explicit 
instruction (M = .83). As expected, observations conducted in comparison classrooms indicated 
comparison teachers rarely used lesson maps (M = .11) and instructional templates (VM = .08) 
during instruction. Although treatment diffusion was minimal, some comparison teachers did 
have access to intervention materials. Table 1 presents a summary of fidelity data, indicating the 
extent to which (a) ECRI materials were used in treatment and comparison classrooms (items 1- 
2), (b) instruction provided in each of six areas of reading was explicit (items 3-8), and (c) 
instruction in treatment classrooms was consistent with the targets of the ECRI intervention 
(items 9-11). 

Student Assessment Measures and Procedures 

Prior to testing students in the fall, data collectors attended three days of standardized 
assessment training. Across the winter and spring, data collectors attended four days of 
additional training. For individually administered student measures, assessment coordinators 
evaluated inter-rater agreement by shadow scoring with data collectors and providing feedback 


on test administration. Inter-rater agreement is reported for assessments in the paragraphs below. 
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Dynamic Indicators of Basic Early Literacy Skills (DIBELS). DIBELS assessments 
were used to measure phonemic decoding skill (i.e., the alphabetic principle) and passage 
reading fluency. The Nonsense Word Fluency (NWF) and Oral Reading Fluency (ORF) 
measures were administered in the fall, winter, and spring. Trained data collectors administered 
DIBELS assessments in all districts. Average inter-rater agreement was 97.7% (range = 90- 
100%) for NWF and 97.4% (range = 91-100%) for ORF across the study. 

NWF. NWF (Kaminski & Good, 1996) is an individually administered, one-minute, 
timed measure of student skill in reading consonant-vowel and consonant-vowel-consonant 
pseudowords. Students can either provide the sounds of the individual letters or the pseudoword 
as a unit. The score for Correct Letter Sounds (NWF-CLS) was obtained by counting the number 
of correct letter sounds students provided. The score for Words Read Correct (NWF-WRC) was 
obtained by counting the number of non-words students recoded correctly. NWF alternate-form 
reliability coefficients range from .67 to .80, and concurrent validity coefficients with readiness 
subtests of the Woodcock-Johnson Psycho-Educational Test ranged from .35 to .55 (Good & 
Kaminski, 2002). 

ORF. DIBELS ORF (Good & Kaminski, 2002) is an individually administered, one- 
minute, timed measure of student skill in accurately and fluently reading connected text. The 
number of words read correctly in one minute is the student’s score on a single passage. To 
determine a student’s benchmark score, the student is administered three grade level passages at 
a single benchmark assessment time point during the school year (beginning, middle, or end) and 
the median score is recorded. In the beginning grades, ORF alternate-form reliability coefficients 
range from .89 to .94 and test-retest reliability coefficients range from .92 to .97 (Good & 


Kaminski, 2002). 
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Woodcock Reading Mastery Test - Revised (WRMT). The WRMT (American 
Guidance Service, 1998) is a standardized, comprehensive battery of tests that measures multiple 
aspects of reading ability, including comprehension, word recognition, and word analysis. Two 
skill clusters comprised of four subtests were administered to students and used in this analysis: 
the Basic Skills Cluster (WRMT-BS), which includes the Word Identification and Word Attack 
subtests, and the Total Reading (WRMT-TR;; full scale), which includes the Word Identification, 
Word Attack, Word Comprehension, and Passage Comprehension subtests. The total standard 
score, derived from grade-based norms, was used in all analyses. As reported in the testing 
manual, the correlation between scores on the WRMT-TR and the Woodcock-Johnson Psycho- 
Educational Battery, Total Reading Full Scale is .88 in Grade 1. Internal consistency for the 
subtests ranges from .94 to .98 in Grade 1. We administered Form H of the WRMT in the fall 
and spring. Average inter-rater agreement between data collectors across the study was 97.8% 
(range = 93 — 100%). 

SAT10. The SAT10 (Harcourt Educational Measurement, 2002) is a group administered, 
norm-referenced test of reading proficiency. Total standard scores, derived from grade-based 
norms, were used in all analyses. The SESAT 2 was used in the fall and the Primary | was used 
in the spring of Grade 1. The test manual indicates that Kuder-Richardson reliability coefficients, 
a measure of internal consistency, are .94 for SESAT 2 and .97 for Primary 1. Also, SESAT 2 
and Primary 1 Total Reading scores are moderately correlated with Otis-Lennon School Ability 
Test, 8th Edition, Total scores (r = .68 and .61, respectively). Trained data collectors 
administered the SAT10 in the fall and spring of first grade to all students participating in the 
project. Subtests administered included Sounds and Letters, Word Reading, and Sentence 


Reading (SESAT 2), or Word Study Skills, Word Reading, Sentence Reading, and Reading 
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Comprehension (Primary 1). Total SAT10 testing time ranged from 110 to 155 minutes. Results 
of the fall administration were used to assign students to Tiers, while spring results served as the 
primary reading outcome measure in the study. 
Observations 

Classroom Observations of Student Teacher Interactions (COSTI). The classroom 
observation protocol used in the study was a modified version of the Classroom Observations of 
Student-Teacher Interactions (COSTI; Smolkowski & Gunn, 2012), which is used to record 
contextual information and quantify student-teacher interactions during instructional activities. In 
the modified version, documented context codes include group size (i.e., whole group, small 
group, and number of students), content codes (i.e., phonemic awareness, alphabetic principle, 
word reading, reading connected text, vocabulary, comprehension, non-reading content, and 
other), curriculum materials used, activity start time, and activity stop time. Similar to the 
original COSTI, observers document instances of teacher demonstrations, or models, student 
practice, and student errors. However, the modified COSTI is used to document three types of 
independent practice: individual responses (i.e., a response by a single student initiated by the 
teacher), group responses (1.e., response by two or more students initiated by the teacher), and 
covert responses (i.e., a non-verbal response by one or more students where the accuracy of the 
response cannot be verified). The modified COSTI is also used to record two types of teacher 
feedback: Observers recorded immediate confirmatory and corrective academic feedback 
delivered by the teacher in response to student practice opportunities. Frequencies of teacher and 
student behaviors are coded within blocks of time, allowing calculation of a rate of student- 
teacher interactions for each observation, irrespective of content code. Rates reported in this 


study represent the number of behaviors observed per minute during a single observation, 
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averaged across observations conducted during the year. Research on the use of the COSTI 
indicates the rate of student-teacher interactions can be used to predict early reading achievement 
(Smolkowski & Gunn, 2012). 

Observation Training and Reliability. Observations were conducted during core 
reading instruction in all ECRI treatment and comparison classrooms in winter and spring. 
Unless schools reported classrooms were providing less than 90 minutes of core reading 
instruction, all observations were scheduled for at least 90 minutes. On average, reading 
instruction was observed for 99.86 minutes in ECRI treatment classrooms (range = 80.50 - 
146.50) and 91.17 minutes in comparison classrooms (range = 74.50 - 114.50). In total, 42 first 
grade classrooms located in three school districts were observed two times during the school year 
using the COSTI instrument. 

At the beginning of each observation round, independent observers received three days of 
training to use the observation protocol and were required to complete a training reliability 
checkout to document inter-rater agreement with the observation coordinator. Inter-rater 
agreement was calculated as the number of coding agreements divided by the total number of 
behaviors coded. Average training reliability across the year was .74. Observers were also 
required to complete a field reliability checkout and meet an inter-rater agreement standard of 
.80 before observing independently at each observation round. Average field reliability across 
the year was .94. To examine maintenance of reliability throughout each observation round, at 
least 20% of all observations were paired. For the paired observation, we estimated the variation 
in scores within and between observation occurrences using an unconditional multilevel model 
and computed the intraclass correlation coefficient (ICC) as the proportion of between reliability 


pairing variance. Based on 31 occasions in which two observers collected COSTI data 
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simultaneously, moderate to high interobserver reliability was obtained for documenting rates of 
teacher models, group practice, individual practice, covert practice, errors, and feedback (ICCs = 
74, .94, .62, .70, .54, and .79, respectively; Landis & Koch, 1977). 

Statistical Analysis 

We assessed intervention effects on each of the primary outcomes with a mixed-model 
(multilevel) time by condition analysis (Murray et al., 2006). This tests differences between 
conditions on change in outcomes from the fall of first grade (T1) to the winter or spring of first 
grade (T2). The specific model tests time coded 0 at T:1 and | at T2, condition coded 0 for control 
and 1 for ECRI, and the interaction between the two. With only 16 schools, tests of time by 
condition used 14 degrees of freedom. The analyses included only students at risk for reading 
difficulty. To address attrition, we used a similar multilevel model. 

This analysis approach included all available data—whether or not students’ scores were 
present at both time points—to estimate differences between assessment times and between 
conditions. The nested time by condition analysis accounts the intraclass correlation associated 
with multiple students nested within the same schools. As a test of net differences, it also 
provides an unbiased and straightforward interpretation of the results (Cribbie & Jamieson, 2000, 
Fitzmaurice, Laird, & Ware, 2004). 

Model estimation. We fit models to our data with SAS PROC MIXED version 9.1 (SAS 
Institute, 2009) using restricted maximum likelihood (REML), generally recommended for 
multilevel models (Hox, 2002). From each model, we estimated fixed effects and variance 
components. Maximum likelihood estimation for the time by condition analysis allows the use of 
all available data and provides potentially less biased results even in the face of substantial 


attrition, provided the missing data were missing at random (Schafer & Graham, 2002). In the 
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present study, we did not believe that attrition or other missing data represented a meaningful 
departure from the missing at random assumption, meaning that missing data did not likely 
depend on unobserved determinants of the outcomes of interest (Little & Rubin, 2002). For 
example, some students were ill at the time of the posttest assessments and others transferred to a 
new school because their family moved. 

The models assume independent and normally distributed observations. We addressed the 
first, more important assumption (van Belle, 2008) by explicitly modeling the multilevel nature 
of the data. Regression methods have also been found quite robust to violations of normality and 
outliers have a limited influence on the results in a variety of multilevel modeling scenarios 
(Bloom, Bos, & Lee, 1999; Donner & Klar, 1996; Fitzmaurice et al., 2004; Maas & Hox, 2004; 
2004b; Murray et al., 2006). This feature of multilevel models also eases concerns about the use 
of different scoring methods used for different measures in the analyses (e.g., raw scores, scaled 
scores, standard scores, etc., available for the WRMT & SATIO). 

Effect sizes. To ease interpretation, we computed an effect size, Hedges’ g (Hedges, 
1981), for each fixed effect. Hedges’ g, recommended by the What Works Clearinghouse 
(WWC, 2008), represents an individual-level effect size comparable to Cohen’s d (Cohen, 1988), 
except that Cohen’s d uses the sample standard deviation while Hedges’ g uses the population 
standard deviation (Rosnow & Rosenthal, 2008). 

Results 

Table 2 presents means, standard deviations, and sample sizes for each measure by 
assessment time and condition. Next, we tested the sample for attrition effects, and differential 
attrition in particular. Finally, we addressed our research hypotheses with the nested time by 


condition model previously defined. Complete model results can be found in Table 3. 
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Attrition 

Participant attrition, also called experimental mortality, can pose a threat to both external 
and internal validity of a study (Barry, 2005; Shadish, Cook, & Campbell, 2002). Student 
attrition was defined as students with data at Ti but missing data at T2. 

Scores for the DIBELS measures of NWF-CLS, NWF-WRC, and ORF and for WRMT 
were missing for 10.5% of students, with 10.2% of the students missing data in control schools 
and 10.8% missing in intervention schools, and the proportion missing did not depend on 
condition for any of these measures (y2 = 0.03, df = 1, p = .8674). For the SAT10, we 
experienced 8.2% attrition at T2, with 10.2% of the students missing data in control schools and 
5.8% missing in intervention schools (y2 = 1.67, df = 1, p = .1964). Thus, attrition rates did not 
differ between conditions. 

Although differential rates of attrition are undesirable, differential scores by condition on 
literacy measures present a far greater threat. We conducted analyses to test whether student T1 
scores on NWF, ORF, SAT10, WRMT-BS, and WRMT-TR differed by attrition, condition, or 
the interaction. The analysis used a mixed-model ANOVA, which nests students’ T1 scores 
within schools and condition. We found no statistically significant interactions between attrition 
and condition. 

Impact on DIBELS 

For students at risk for reading difficulty, we expected students exposed to ECRI to 
greater gains on NWF-CLS, NWF-WRC, and ORF from fall to winter and fall to spring of first 
grade. Table 2 presents the means and standard deviations for DIBELS measures collected in the 
fall, winter, and spring. 


The time by condition analysis demonstrated that students at risk for reading difficulties 
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in ECRI schools made statistically significant gains from fall to winter over those in control 
schools on NWF-WRC (t = 2.55, df = 14, p = .0230) and ORF (t = 2.24, df = 14, p = .0415). 
Students in ECRI schools, when compared to controls, made marginally significant gains on 
NWF-WRC (t = 1.81, df = 14, p = .0919) fall to spring but not on NWF-CLS or ORF. 
Impact on SAT10 and WRMT 

We next tested whether students at risk for reading difficulty performed better in ECRI 
schools than control schools on fall-to-spring gains on the SAT10 and WRMT. The analyses did 
not result in statistically significant findings for SAT10, WRMT-BS, or WRMT-TR. We found a 
marginally significant gain, however, for WRMT-BS (t = 2.00, df = 14, p = .0651). 

Practice Opportunities and Condition 

Finally, we examined whether relationship between condition and literacy outcomes was 
altered by the rate of independent group practice opportunities. We found an increasing 
difference between conditions in gains on ORF from fall to winter as the rate of group practice 
increased across classrooms (t = 2.55, df = 14, p = .0230). Intervention schools performed 
statistically significantly better than comparison schools when their teachers offered group 
practice at a rate of a rate of approximately 1.8 opportunities per minute or better, as shown on 
Figure 1. 

Figure | depicts the interaction between treatment condition and independent group 
practice opportunities for the oral reading fluency (ORF) dependent variable. The vertical axis 
shows the difference between conditions, intervention minus control, and the horizontal axis 
depicts the rate of independent group practice opportunities. The figure, then, shows the 
difference between conditions across the range of classrooms characterized by the rate of group 


opportunities for practice. The increasing slope of the line shows that as classrooms increase the 
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rate of practice they provide to students, the ECRI intervention produces greater differences. 
The two lighter lights show the 95% confidence bounds on the difference between conditions. 
At the lower end of the range, in terms of rate of practice, the 95% confidence bounds include 
zero (no difference between conditions), which implies that two conditions do not differ. Once 
the rate of practice reaches 1.8 attempts per minute, the lower of the 95% confidence bounds 
crosses the zero, which implies that the ECRI classrooms perform better than controls at that 
point. The analyses did not produce statistically significant interactions for the remaining 
dependent variables. 
Discussion 

The primary objective of the study was to evaluate the efficacy of a highly specified, 
tightly aligned multi-tiered reading intervention to improve the reading achievement of first 
grade at-risk students relative to a business-as-usual RtI condition. We examined student gains 
for proximal and distal outcomes from fall-to-winter and fall-to-spring. For students at risk for 
reading difficulty, we expected greater gains for ECRI intervention students on the both proximal 
and distal measures, as compared to students in the comparison condition. We also expected 
students who participated in the ECRI intervention to demonstrate greater gains on proximal 
outcomes from fall to spring of first grade, than for distal outcomes. Finally, we also examined 
whether the relationship between condition and literacy outcomes was altered by the rate of 
group practice opportunities. We hypothesized that the highly controlled and deliberate practice 
(Ericsson, 2008) in the treatment condition would lead to stronger fluency gains relative to less 
controlled practice in the comparison condition. In the following sections we summarize results 
of the study, present implications, describe limitations of the study, and conclude with some 


steps for future research. 
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The nature of the counterfactual in this study highlights the significance of these finding 
and should guide the interpretation of the significant impacts on proximal measures as well as 
the non-significant, yet substantive effect sizes for distal outcomes. All of the schools recruited 
for the study were ostensibly implementing a multi-tiered approach to reading instruction. All of 
the schools had a 90 minute reading block in which they implemented a core reading program, 
used a screening measure to identify students for Tier 2 support, and had a 30 minute small 
group Tier 2 intervention block. In addition, due to blocking on district (and functionally on core 
reading program), treatment and control schools within district were using the same core reading 
program and materials. Thus, the nature of the RCT tested the effect of a multi-tiered 
intervention that was highly specified, with increased explicitness and intensity, and with highly 
aligned and cohesive Tier 1 instruction and Tier 2 intervention support relative to the control 
condition. Additionally, these features were embedded within a PD and coaching model that 
provided a deep and precise focus on how classroom teachers and interventionists were 
implementing the multi-tiered intervention features. Thus, the intervention effects were not 
derived from a study with a no treatment control, rather the control group was implementing a 
research —based, tiered system of support for their first grade students. The key difference was 
the degree of specificity, explicitness and alignment that comprised the ECRI intervention. 
Results Summary 

First, we hypothesized that at-risk readers in the ECRI intervention would demonstrate 
stronger gains on all outcomes, when compared to at risk readers in the business-as-usual multi- 
tiered comparison condition. Compared to control students, we also expected stronger gains for 
at risk readers in the intervention group on proximal measures relative to distal measures. 


Analysis of proximal measures revealed a statistically significant difference favoring 
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intervention students on NWF-WRC and ORF fall-winter, with effect sizes of g = +.42, and 
+.34, respectively. There were no significant effects for NWF-WRC, NWF-CLS or for ORF fall- 
spring, however, there were substantive effect size differences favoring the treatment condition 
on NWF WRC and ORF fall-spring (g = .38 and +.30, respectively). The What Works 
Clearinghouse describes non-significant findings with effect sizes larger than g = +.25 as 
substantive and rate the effectiveness of a particular intervention as “potentially positive”. 

The magnitude of the effects on decoding outcomes is comparable to those found in 
previous studies of effective Tier 2 interventions (Ebaugh, 2000; Gunn et al., 2000; Jenkins 
et al., 2004; Lennon & Slesinski, 1999; Vadasy, Sanders, & Peyton, 2005). However, the 
significant and substantial effect on ORF is fairly unique and has only been demonstrated in one 
previous study (Gunn, 2002). It is also interesting to consider the attenuating of the fluency 


effect from the fall to winter to the fall to spring analysis. One interpretation is that the efficacy 
of the intervention lessened as the year progressed. Furthermore, the lack of significant effect 
(though substantive effect size) could either be a result of statistical power or, conversely an 
ephemeral effect. However, an alternative explanation, and one supported empirically has to do 
with the developmental nature of fluency development. There are several recent studies that have 
demonstrated nonlinear growth on ORF measures, with the greatest amount of growth occurring 
in the fall season (Baker et al., 2008; Christ et al., 2010; Cummings ###). Thus, it may be 
reasoned that the ECRI intervention was not less effective in the spring season; rather growth is 
typically stunted in the spring season. Even so, the question of non-significance still remains 
(i.e., low power versus ephemeral effect), and further replication is warranted. 

Analysis of distal measures revealed non-significant effects for WRMT-BS, SAT10 and 


WRMT-TR. However, the magnitude of the effect size on each measure is substantive and 
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represents a “potentially positive” effect. On the WRMT-BS the impact was commensurate with 
the effect on foundational reading proximal measures (g = +.45). Of note was the strong trend 
favoring the intervention group on the SAT-10 and WRMT-Total Reading scaled scores. 
Statistical tests indicate robust effect size differences between conditions, albeit non-significant, 
on these two distal indicators of general reading achievement, one a group administered (SAT- 
10; g = +.43) and the other an individually administered measure (WRMT-TR; g = +.44). 

Unfortunately, our study did little to disentangle the mixed results demonstrated to date 
on the importance (or lack thereof) of curricular alignment across Tiers and 2. Our findings are 
generally similar to the findings in the Chambers et al (2011) study such that a multi-tiered 
intervention that has strong alignment between Tiers | and 2 appears to contribute to stronger 
results for at risk readers. However, our findings are in stark contrast to the Scanlon et al (2008) 
study in that in their study there were no significant differences between the condition that had a 
strong alignment between Tiers | and 2 (in the Combined PD and Intervention Condition both 
based on Interactive Strategies Approach; ISA) and the other conditions that did not have strong 
alignment between Tiers 1 and 2. 

It is interesting to note that the approach that Scanlon et al implemented substantially 
differed from the ECRI approach in terms of level of specificity provided to teachers and 
interventionists. The ECRI intervention includes carefully controlled teacher wording, a set 
scope and sequence of skills and concept instruction, carefully planned practice opportunities for 
teachers to deliver students, and specific protocols for providing corrective feedback to students. 
The ISA approach alternatively relies on PD and recommendations for teachers to use a variety 


of techniques and strategies to work with students. There is no teacher manual or standard 


protocol top follow. It is also likely that the ISA approach varied in level of explicitness and 
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intensity given the teacher or interventionist had much more control to ###. Thus, it appears 


that the combination of specificity and explicitness contributed to the range of positive outcomes 
in the current study. Additionally, the mixed results in the research literature on curricular 
alignment may be somewhat related to what is actually being aligned. 

Our second hypothesis was related to the role of student practice in the development of 
fluent reading skill. Specifically, we hypothesized that the highly controlled and deliberate 
practice in the treatment condition (e.g., ensuring student were practicing building fluency in 
appropriate texts, including immediate corrective feedback) would lead to stronger fluency gains 
relative to less controlled practice in the comparison condition. Accordingly, we expected an 
interaction between treatment condition and the rate of student practice opportunities. We found 
interactions between condition and rate of practice opportunities for ORF gains fall-to-winter. In 
other words, when teachers offered group practice at a rate of 1.8 opportunities per minute or 
higher, intervention schools performed better than comparison schools. We did not find effects 
on the other proximal and distal measures. This interaction was not the result of the intervention 
classrooms scoring greater ORF scores as you moved up the rate of practice distribution, but as a 
result of the control classrooms demonstrating a greater decrease in ORF scores as you increased 


the rate of practice distribution. In other words, the more practice that control classroom 


conducted, the lower their ORF scores. This finding aligns with our hypothesis that practice that 
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Study Implications 

The findings from the current study have potential implications for a number of education 
stakeholder groups, though we temper our interpretation until further replication studies are 
conducted. We contend that the current study offers preliminary evidence that school and district 
decision makers responsible for the implementation of multi-tiered instruction and intervention 
models might consider three elements when choosing a model to support their struggling 
learners: the level of specificity provided to teachers and interventionists to implement the 
model, the level of explicitness and intervention intensity for students, and finally, the degree of 
alignment between Tier | instruction and Tier 2 intervention materials. In the context of the 
current study, we manipulated the design and delivery of Tier | instruction to be more accessible 
for students that were below grade level for reading achievement. As a result, intervention 
teachers provided more explicit instruction, incorporated more frequent and deliberate practice 
opportunities, and provided more precise and immediate corrective feedback compared to control 
teachers (authors names omitted). However, as we revisit in the limitations section, we cannot 
for certain make any statement about any particular component we believe to be essential, rather 
we can say that when these components are brought together, these components can accelerate 
student learning relative to the business-as-usual multi-tiered control group. 

Increasing the likelihood of teachers implementing research-based strategies in authentic 
school settings is a major goal of education leaders. Likewise, decreasing the variability of 
instruction practices and increasing fidelity of implementation to models of instruction and 
intervention is particularly difficult (Gersten, Chard, & Baker, 2000; Gresham, MacMillan, 
Beebe-Rankenberger, & Bocian; 2000). To address these issues in the context of ECRI, we 


developed highly specified lesson plans and teaching routines to support standard 
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implementation of instruction and intervention materials. Our goal was to increase the level of 
specificity to ensure that teachers provided students with explicit and, when appropriate, 
intensive instructional supports (i.e. in the context of both Tier | and Tier 2). These routines 
provided clear expectations to teachers for what content to cover during instruction and 
intervention lessons and highly specified guidance for explicit and engaging teacher- student 
interactions. Akin to the Checklist Manifesto (Gawande, 2009), the goal of the specified routines 
was to increase the degree to which practitioners implement evidence-based practices with 
fidelity and integrity. 

The approach of using highly specified instruction and intervention routines can also be 
used as a tool for coaches and school leaders to define and measure implementation fidelity and 
to provide subsequent implementation goals for teachers. It is important to note that school based 
personnel (rather than researchers) delivered both the Tier 1 portion and the Tier 2 portions of 
the model. Having school personnel as implementers, notably a unique feature of this study, 
increases the external validity of the study’s results. 

The study findings also have potential implications for publishers and developers of core 
reading programs and tier 2 interventions. First, in our opinion, the degree of specificity and 
guidance provided to teachers for delivering explicit instruction in current reading programs is 
lacking. Many programs do not provide enough explicit, scaffolded instruction or practice 
opportunities for learners at risk of reading difficulty (Gersten, 1999). Second, core program and 
intervention developers and publishers should strive to align instruction and intervention 
materials to ensure struggling students are delivered a robust and coherent tiered support plan. 
Most publishers of core programs also offer Tier 2, small group intervention materials. However, 


often different authors with competing pedagogical approaches develop them or they are very 
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superficial in the level of intervention intensity (Baker, Fien, & Baker, 2010). Even when the 
programs are philosophically aligned, the misalignment of scope and sequence of skill 
introduction, structured practice opportunities, and review cycles between Tier 1 and Tier 2 
materials can compromise a coherent reading plan within and across each day of instruction for 
struggling learners. 
Limitations and Future Directions for Research 

One of the primary limitations of the ECRI study is that we are not able to determine the 
relative contribution of different features or components of the multi-tiered intervention on 
student achievement. Given the scope of the intervention, the nature of the research design, and 
the funding priorities set forth in the IES Special Education Systemic Intervention grant program, 
we can only determine and discuss the net effect of the whole ECRI system on student 
achievement. This does not minimize the value or significance of the findings of the current 
study; however, it does raise a number of questions related to the relative value for each of the 
components of the system. For example, did the professional development and in depth 
classroom coaching have a similar impact on student achievement as the instructional design 
templates that made classroom instruction more explicit and systematic? Additionally, how much 
of a boost in Tier 2 student gains were as a result of the daily small group Tier 2 intervention? 
We cannot answer these and many other important questions in the context of the current study. 
However, follow-up component analyses studies could help determine the relative merit of 
different components in the multi-tiered intervention. 

A second limitation is related to our relatively modest sample size (School N = 16) and 
larger than expected school level ICC’s for our outcome variables (Hedges & Hedberg, 2008). 


This particular point has bearing on our non-significant outcomes with relatively large effect 
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sizes (1.e.., range g = +.27 - +.45). A subsequent study with more power (i.e, more schools) 
could help us determine if these no significant effects are truly substantive in nature or merely 
fleeting. Additionally, the schools were sampled from one state in the Pacific Northwest that has 
received a lot of statewide support for Rti and Multi-tiered models. We are not sure if the 
positive effects generated in the current study would replicate in states where there is more or 
less support. To address these two limitations we are currently conducting a larger replication 
study with three times the amount of schools and in more diverse geographical settings. 
Conclusion 

In summary, we found significant effects on three of the outcome variables in the area of 
word reading, decoding, and fluency for the ECRI intervention. The intervention also had 
substantive effects on reading achievement. These findings are particularly compelling given the 
nature of the counterfactual in that control schools were implementing the same core reading 
programs and a similar tiered approach to reading instruction. However, further studies that 
assess the relative value of intervention components and planned variations of intervention 
components are necessary. Additionally, an examination of the potential for the replication of 


intervention effects in more diverse geographical settings is warranted. 
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Table 1 
Fidelity of Implementation by Intervention Condition 


Comparison Intervention 

(n = 24) (n = 18) 

M (SD) M (SD) 
1) Lesson maps used 11 (.16) .90 (.16) 
2) Card templates used 08 (.13) 91 (.16) 
3) Instruction in phonemic awareness 17 (.28) 85 (.19) 
4) Instruction in alphabetic principle 62 (.26) .94 (.13) 
5) Instruction in word reading .69 (.30) .90 (.16) 
6) Instruction in reading connected text 87 (.17) 85 (.21) 
7) Instruction in vocabulary 50 (.33) .60 (.31) 
8) Instruction in comprehension strategy 3925) JT? (33) 
9) Consistent with intentions and design -- 83 (.21) 
10) Activities in maps were executed -- .76 (.16) 
11) Lesson was complete -- 73 (.29) 
Explicit instruction2 57 (.16) .78 (.16) 
ECRI specific fidelity summary score3 -- 83 (.15) 


Note. M = mean; SD = standard deviation. 1 Fidelity items were rated as 0 (no), 0.5 (partially), or 
1 (yes). 2 The explicit instruction score represents the mean across items 3 through 8. 3 The ECRI 
specific fidelity represents the mean across items 1, 2, 9, 10, and 11. Items and scale scores were 
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averaged across the three observation occasions. 
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Table 2 


Descriptive Statistics for All Measures by Time and Condition 


Fall Winter Spring 
Measure ECRI Control ECRI Control ECRI Control 
Nonsense Word M 33.5 35.9 58.02 54.8 75.2 73.6 


Fluency, Correct 
Letter Sequences (SD) (13.2) (15.8) (22.3) (20.1) (29.1) (29.0) 


N 116 143 115 138 107 132 
Nonsense Word M 7.7 74 14.4 10.1 1S 14.9 
Fluency, Words 
Recoded (SD) = (4.8) (5.0) (10.3) (8.9) (12.6) (12.9) 
Correctly 
N 116 143 115 138 107 132 
Oral Reading M 10.1 11.2 30.6 26.7 60.2 56.5 
Fluency 
(SD) (6.8) (7.5) (18.1) (14.7) (22.2) (23.8) 
N 116 143 115 138 107 132 
SAT-10 Total M 467.0 468.4 - - 553.0 543.7 
Reading Scaled 
Score (SD) (11.0) (11.2) - - (32.8) (33.5) 
N 120 147 - - 113 132 
WRMT Basic M 105.4 105.0 - - 112.8 109.3 
Skills Cluster 
Standard Score (SD) (6.6) (7.0) - - (7.8) (8.3) 
(Grade Norm) 
N 111 142 - - 107 132 
WRMT Total M 99.7 99.9 - - 109.2 106.7 
Reading Cluster 
Standard Score (SD) = (7.4) (7.5) - - (7.4) (7.7) 
(Grade Norm) 


N 111 142 - - 107 132 
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Table 3 


Estimates from Mixed-Model Time by Condition Analysis to Test Condition Effects for ECRI Schools Compared to Control Schools 


Fall to Winter Fall to Spring 
Effect or Statistic NWF-CLS NWF-WRC — ORF NWF-CLS NWF- ORF SATIO WRMT-BS WRMT-TR 
WRC 
Fixed Intercept’. -35.90"** “/587** 11.3672" “36.1 BFF" 7O8**F >) “1152 *** AGS Al e* «104 890%"" OO Are 
Effects (3.02) (1.06) (1.44) (3.61) (1.35) (2.38) (4.34) (1.23) (1.37) 
Time 18.577** 202" LASOF** .3612"**. “F22*: AC OGFe* J302F* 1 3.90%* 6.47% *% 
(2.28) (1.04) (1.68) (4.31) (1.82) (3.32) (5.62) (1.25) (1.39) 
Condition -3.53 -.03 -1.32 -3.70 -.07 -1.68 -2.61 25 -.27 
(4.41) (1.56) (2.14) (5.28) (1.99) (3.50) (6.31) (1.80) (1.99) 
Time x 5.61 3.93% ao" 6.36 4.83+ 6.92 14.19 3.65+ 3.30 
Condition (3.36) (1.54) (2.46) (6.30) (2.67) (4.85) (8.17) (1.82) (2.01) 


Variances Residual 148.77*** 35.60*** = 75.72*** = 301.93*** 64.39*%** 174,92*** 373.95*** D272. 91***  18.44%%% 
(13.55) (3.24) (6.95) (28.14) (3.95) (16.42) (34.02) (2.20) (1.78) 


Student TSA Ale? 1610") ~ PA TTte IAT DOFee: AI ZAS OBL BLES. A GOS 24 O42 OSs ee 
(20.13) (3.53) (10.77) (30.70) (5.39) (17.74) (31.91) (3.61) (3.59) 


School 44.41; 3.67 1.28 2242 62 <2 19.33 4.51 5.69 
Intercept (25.43) (2.88) (5.07) (30.51) (4.05) (13.13) (42.99) (3.86) (4.77) 


School 11.77 2.21 6.62 54.90; 9.19 32.86; ~—«:102.83* 4.733 6.48% 
Gains (755) (1.66) —s (4.62) (28.89) (5.08) (18.22) ~—«- (50.08) ~—s (2.45) _~—Ss«(3.06) 
ICC 073 059 080 154 125 158 216 471 260 
Hedges’ g Time x 265 ALS 339 219 379 300 428 AS2 437 


Condition 


Note. Table entries show parameter estimates with standard errors in parentheses. Time was coded 0 for T: and 1 for T2. Condition 
was coded 0) for control and | for ECRI. All tests fixed effects used 14 df. 


+p<.10.* p< .05. ** p< .01. *** p < .00 
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